This Page Is Inserted by IFW Operations 
and is not a part of the Official Record 



BEST AVAILABLE IMAGES 



Defective images within this document are accurate representations of 
the original documents submitted by the applicant. 

Defects in the images may include (but are not limited to): 



BLACK BORDERS 

TEXT CUT OFF AT TOP, BOTTOM OR SIDES 
FADED TEXT 
ILLEGIBLE TEXT 
SKEWED/SLANTED IMAGES 
COLORED PHOTOS 

BLACK OR VERY BLACK AND WHITE DARK PHOTOS 
GRAY SCALE DOCUMENTS 



IMAGES ARE BEST AVAILABLE COPY. 



As rescanning documents will not correct images, 
please do not report the images to the 
Image Problem Mailbox. 



O P I c 

Office de la propriety 

INTELLECTUELLE D Lf CaNaOA 




(12) 



C I P o 

Canadian Intellectual 
Property Office 



(is) (ca) Demande-Application 



(72) LE BLANC, Andrew D., CA 
(71) MIND MELD MULTIMEDIA INC., CA 
(5i)Int.Cl 6 G06F3/14 
(30) 1997/03/14 (08/818,236) US 

(54) INTERFACE UTILISATEUR GRAPHIQUE ET APPAREIL 

UTELISANT CETTE INTERFACE 
(54) GRAPHICAL USER INTERFACE, APPARATUS AND METHOD 



(2D (ad 2,231,699 

(22) 1998/03/10 
(43) 1998/09/14 




/o 



(57) L'invention est une interface utilisateur graphique 
pour programmes informatiques qui comprend la 
representation d'un visage (un avatar) sur un ecran 
d'ordinateur, lequel permet a un utilisateur de 
communiquer son attitude en rapport avec une situation 
presentee par un programme informatique. L/expression 
du visage change quand Tutilisateur deplace le curseur 
sur le visage. En reponse a une situation affichee sur 
l'ecran, Tutilisateur choisit l'expression du visage qui 
correspond a son attitude. La situation affichee change 
en consequence. 



(57) A graphical user interface for a computer program 
includes the representation of a face (an avatar) on a 
computer screen which allows a user to communicate the 
user's attitude to a situation represented by the computer 
program. The expression of the face changes according 
to the user's movement of a cursor over the face. In 
response to a situation appearing on the computer screen, 
the user sets the expression on the face to correspond 
with the user's attitude. The situation on the computer 
screen changes accordingly. 
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GRAPHICAL USER INTERFACE, APPARATUS AND METHOD 

Field of the Invention 

The invention relates to graphical user interfaces 
and is particularly concerned with graphical user interfaces 
in computer programs. 

Background of the Invention 

Pictorial -based interfaces in computer programs are 

10 well-known. For example, U.S. patent No. 4,752,069 issued to 

Okada in 19 88 allows a video game user to change the movement 
of a character on the video screen by shifting a lever in the 
desired direction. An arrow on the computer screen indicates 
the direction in which the lever ought to be shifted in order 
to effect successful movement of the character on the screen. 
Since the arrow simply tells the user how to behave in order 
to succeed in the game, the game does not require any 
independent thought on the part of the user. 

U.S patent No. 5,452,414 issued to Rosendahl in 1995 

20 relates to a manipulable icon. Icons are typically two 

dimensional and display information about aspects of a 
computer program. The icon disclosed in this patent is 
notionally three dimensional and can be manipulated to show 
different information about an application such as the name of 
a document, its size, its creator, copyright information, etc. 
The icon described in this patent acts as a vehicle to 
transmit information about one or many documents to the user. 
The icon does not otherwise help the user interact with a 
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particular computer application. In other words, while the 
icon provides information to the user, it does not help the 
user communicate information to a computer application. 

U.S. patent No. 5,498,002 issued to Gechter in 1996 
describes an interactive, electronic game having, among other 
aspects, "character behaviour controllers". The patent 
describes a game in which a user can choose a character and 
then control its behaviour according to the character's 
"character controller logic". For example, when a character 

10 is selected, a pop-up window may appear allowing the user to 

modify the character's actions or behaviour by selecting 
desired options from the pop-up menu. In other words, the 
user controls the character's behaviour directly by inputting 
specific commands. The characters simply behave as instructed 
by the user. The characters do not react to the user's 
behaviour or attitude to a particular situation. 

There is at least one game available in which the 
characters behave according to or in response to a user's 
mood. In other words, the user does not tell the characters 

20 how to behave. Rather, in the game "Mode", produced by the 

company Animatics and available on the World Wide Web on the 
Internet, the user indicates his or her mood to a particular 
situation in the game, and the characters in the game react 
accordingly. The user indicates his or her mood by moving the 
cursor on the "Mood Bar" depicted on the computer screen and 
clicking the mouse. The Mood Bar is a rectangle showing a 
spectrum of colours, with red on the left hand side and green 
on the right hand side with appropriately varying colours in 
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the middle. The user is meant to indicate his or her mood to 
a particular character or situation in the game by moving the 
cursor on the colour that best represents the user's mood (red 
for angry and green for happy, for example) and clicking the 
mouse. The characters in the game will then react 
appropriately in view of the user's mood. 

While the Mood Bar does allow the game's characters 
to behave or react in response to the user's mood, the game 
does not consider other aspects of the user's personality or 

10 emotions. For example, a user could be in a good mood despite 

negative events that might occur in the game. Accordingly, it 
would be desirable to allow the characters in a computer game 
or program to react to aspects of the user's attitude to a 
situation in the game or program which could change from event 
to event, even though the user's mood may remain the same. 
Further, to reflect real-life situations, it may be more 
helpful or educational to the user to be able to see how his 
or her attitude may affect a given character's behaviour in a 
particular situation . 

2 0 As indicated above, with the Mood Bar, the user 

points the cursor at a colour that represents his or her mood 
and clicks the mouse. However, the Mood Bar fails to provide 
any indication to the user as to the mood just selected or any 
indication that the program has even registered (or taken into 
consideration) the newly selected mood. 
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Summary of the Invention 

It is an object of the invention to obviate or 
mitigate one or more of the above identified disadvantages. 

According to a first broad aspect, the invention 
provides a computer program for use in a computer apparatus, 
the computer apparatus having a visual display device, an 
audio device and an input means, wherein the computer program 
when implemented on the computer apparatus displays on the 
display device a graphical user interface having a changeable 

10 appearance and wherein, in response to information conveyed by 

the computer program on at least one of the display device and 
the audio device, a user, using the input means, sets the 
appearance of the graphical user interface to correspond with 
an emotion of the user and wherein in response to the emotion 
of the user as represented on the graphical user interface, 
the computer program conveys further information on at least 
one of the display device and the audio device. 

According to another broad aspect, the invention 
provides an interactive, electronic apparatus for a user to 

20 communicate the emotion of the user comprising a display 

screen; an audio device; a graphical user interface visible on 
the display screen and having a changeable appearance; an 
input device for the user to set the appearance of the 
graphical user interface to correspond with an emotion of the 
user; a computer program implemented on the interactive, 
electronic apparatus, for conveying information on at least 
one of the display screen and the audio device and wherein in 
response to the emotion of the user as represented in the 
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appearance of the graphical user interface, the computer 
program conveys further information on at least one of the 
display screen and the audio device. 

According to yet another broad aspect, the invention 
provides a method for a user to interact with a computer 
program run on a computer wherein the computer comprises a 
display means, an audio device and an input means and the 
computer program displays a graphical user interface having a 
changeable appearance on the display means, wherein the method 

10 comprises the steps of the computer program conveying 

information on at least one of the display means and the audio 
device; the user, using the input means, setting the 
appearance of the graphical user interface to correspond with 
an emotion of the user; and the computer program conveying 
further information on at least one of the display means and 
the audio device in response to the emotion of the user as 
represented on the graphical user interface. 

Advantages of the present invention include allowing 
a user to interact with characters of a computer program by 

20 having the user express his or her attitucLe to particular 

characters or situations produced by the program and having 
the characters or situation in the program change accordingly. 
It is another advantage of the present invention to provide 
the user with immediate feedback showing a representation of 
the user's attitude. 
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Brief Description of the Drawings 

Preferred embodiments of the invention will now be 
described with reference to the attached drawings in which 

Figures la and lb are representations of a computer 
screen having a graphical user interface depicting a positive 
and negative attitude, respectively, in accordance with an 
embodiment of the present inventions- 
Figures 2a and 2b are state diagrams of graphical 
user interfaces in accordance with an embodiment of the 
10 present invention when the graphical user interface has only- 

three states and only five states, respectively ; 

Figure 3 is a flow chart depicting the process steps 
carried out by the preferred embodiment of the present 
invention; and 

Figure 4 is a decision tree illustrating possible 
decisions at various times during progress of a computer 
program in accordance with an embodiment of the present 
invention where the graphical user interface has three states 
as shown in Figure 2a. 

20 

Detailed Description of the Preferred Embodiment 

The present invention relates to a graphical user 
interface for interactive computer software programs. A user 
manipulates the graphical user interface to reflect the user's 
attitude or other emotion. In view of a user's attitude or 
other emotion as represented by the graphical user interface, 
characters in the program react accordingly. Computer 
programs used in accordance with the preferred embodiment of 
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the present invention are interactive and could be games, 
governmental or corporate employee training software or 
personnel assessment software, for example. Of course, 
inasmuch as software can be incorporated in or as part of 
computer hardware, the invention could also be or form part of 
computer hardware. That said, the preferred embodiment of the 
present invention relates to software, as described below. 

The computer has a microprocessor which controls the 
processes referred to below including the display on the 

10 computer screen. 

Figures la and lb are representations of a computer 
display 10 having a graphical user interface ("GUI") 16 
comprising an avatar 17. An avatar is a representation of the 
user of the computer program. In other words, the avatar is 
dynamic and represents certain changeable attributes of the 
user. The avatar 17 of Figure la represents a user with a 
positive attitude, for example, while the avatar 17 of Figure 
lb represents a user with a negative attitude. The smiling 
face shown in the avatar of Figure la is merely a depiction of 

20 a person with a positive attitude. Of course, it is possible 

for a person to have a negative attitude and smile at the same 
time and, accordingly, more accurate or different 
representations of the user's attitude are, of course, 
possible. . For example, the avatar could be an entire body of 
a person instead of just the face and the user's attitude 
could be depicted by the body language of the avatar or by the 
expression on the face of the avatar in addition to body 
language of the avatar. For example, an avatar with slouched 
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posture or crossed legs could, for example, depict a negative 
attitude, while upright posture and open legs may depict a 
positive attitude . 

The GUI 16 occupies an area of the computer display 
10. The representation of the GUI 16 on the computer display 
16 could consist entirely of an avatar 17, for example, or as 
shown in Figures la or lb, the GUI 16 could additionally 
occupy an area around avatar 17, which in Figures la and lb 
are rectangles. The area of the GUI 16 around the avatar 17 

10 could, however, be of any shape and size. 

The computer display 10 also has an area 12 showing 
a computer program's character (s) 14 in various situations. 
The GUI 16 is manipulated by the user so as to reflect the 
user's attitude (or other emotion) to the character 14 or 
situation currently displayed in area 12. The avatar 17 of 
the GUI 16 in Figure la reflects the user's positive attitude, 
whereas the avatar 17 of GUI 16 in Figure lb reflects the 
user's negative attitude. 

According to the preferred embodiment of the present 

20 invention, the user indicates his or her attitude to a 

situation by first using a computer mouse (not shown) to move 
a cursor 2 0 over the GUI 16. When the cursor is outside the 
GUI 16, the cursor 2 0 appears in a default form, which, as 
shown in Figure la, is in the form of a single-headed arrow, 
although, of course, any other shape for the default cursor 
can be used. When the user moves the cursor 2 0 over the GUI 
16, the shape of the cursor changes its form, which in the 
embodiment shown in Figure lb, is a double-headed arrow, to 
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indicate that the user can now input his or her attitude. . 
When the cursor 2 0 is moved or placed over the GUI 16, the 
user can then input his or her attitude to the current 
situation or character by clicking a button of the mouse and 
holding the button down while moving the cursor 2 0 either 
right or left. If the cursor is moved right, the expression 
on the avatar 17 of the GUI 16 will change to become more 
positive. If the cursor is moved to the left, the expression 
on the avatar 17 of the GUI 16 will change to become more 

10 negative. When the user unclicks the mouse, the current 

expression of the avatar 17 of the GUI 16 reflects the user's 
attitude to the current situation in the computer program, and 
the character (s) 14 in area 12 will react accordingly. 

Of course, instead of the user manipulating a mouse, 
movement of the cursor could also be effected by any of a 
number of possible input or pointing means such as by pressing 
appropriately programmed key sequences of a computer keyboard 
or by using a joystick, track ball or touch pad, for example. 
Alternatively, a cursor may not be necessary at all, where 

2 0 certain keystrokes or a joystick or verbal commands, for 

example, are programmed to control the GUI 16 directly. 

The GUI 16 could be programmed to have or represent 
any number of different possible attitudes. Figure 2a is a 
state diagram of an embodiment of the present invention where 
the GUI 16 has a total of three different attitudes, namely 
negative 22, neutral 24 and positive 26. For example, if the 
GUI 16 currently reflects a neutral attitude 24 and the user 
places the cursor 2 0 over the GUI 16 and clicks the mouse and 
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moves the cursor 20 left, then the GUI 16 will become negative 
22. In other words, the expression on the avatar in GUI 16 
will change to reflect a negative attitude. If the cursor 20 
is now moved to the right, the avatar in GUI 16 will change to 
reflect a neutral attitude 24. If the cursor is moved further 
to the right, the expression on the avatar will change to 
reflect a positive attitude 26. The GUI could be programmed 
to reflect any number of different possible attitudes. Figure 
2b for example, is a state diagram of an embodiment of the 

10 present invention where the GUI 16 has been programmed to have 

five possible different attitudes (ie: more negative, 
negative, neutral, positive, more positive). 

Referring to Figure 3, a flow chart is shown 
depicting the process steps carried out by the preferred 
embodiment of the present invention. When the computer 
program begins, it will likely present an initial situation or 
present a character 14 to the user on an output device, likely 
either visually on the display 10 or possibly aurally, through 
a speaker (not shown) . As shown at "start" block 40, the GUI 

20 16 will likely initially be set to reflect a neutral attitude 

and the cursor 2 0 will have a default appearance. The 
microprocessor will then consider whether or not the user has 
moved the cursor 20 over the GUI 16 as shown at block 42. If 
the cursor 2 0 is not over the GUI 16, then the cursor 2 0 will 
maintain its default appearance as shown at block 44 and the 
microprocessor will repeat the step shown at block 42. If the 
cursor 2 0 is moved over the GUI 16, then the cursor 2 0 becomes 
a double-headed arrow as shown at block 46. The 
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microprocessor now considers whether or not the cursor 2 0 has 
moved off the GUI 16 before the user has clicked the mouse as 
shown at block 48. If the mouse has moved off the GUI 16 
before the user has clicked the mouse, then the microprocessor 
returns to block 42 as shown at block 48. Otherwise, the 
processor considers if the user has clicked the mouse as shown 
at block 50. If the user has not clicked the mouse, then the 
microprocessor returns to block 48. Otherwise, the 
microprocessor considers the movement of the cursor 2 0 as 
shown at block 52. If the cursor 20 is moved left, then as 
shown at blocks 54 and 56, the expression of the avatar 17 of 
the GUI 16 becomes more negative. If the cursor 20 is moved 
right, then as shown at blocks 54 and 58, the expression of 
the avatar 17 of the GUI 16 becomes more positive. If the 
cursor 20 is moved neither to the left nor to the right, then 
as shown at blocks 52 and 60, the expression of the avatar 17 
of the GUI 16 remains unchanged. Moving in the direction of 
the flow from any of blocks 56, 58 or 60, the microprocessor 
now considers if the user has unclicked the mouse, as shown at 
block 62. If the mouse remains clicked, then the 
microprocessor returns to block 52 (to determine if the 
expression of the avatar 17 of the GUI 16 should be further 
modified pending right or left movement of the cursor 2 0 by 
the user prior to unclicking the mouse). Otherwise, the user 
has unclicked the mouse, and as shown at block 64, the 
character in the program reacts in view of the user's attitude 
as reflected in the expression of the avatar 17 of GUI 16. At 
this point, the user must input his or her attitude to the 
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most recent reaction of the character. Accordingly, the 
processor returns to block 42. 

As suggested at block 64, the current character in 
the program may react differently in view of a user's positive 
attitude as opposed to a user's negative attitude. An example 
of a possible implementation of an embodiment of the present 
invention is shown in Figure 4, which is a decision tree 
illustrating possible decisions at various times during 
progress of a computer program in accordance with an 

10 embodiment of the present invention. The decision tree of 

Figure 4 relates to a GUI 16 having three possible states as 
shown in Figure 2a. Of course, if the GUI 16 were programmed 
for more than three possible states, then each node (where a 
decision is being made) of the decision tree would have a 
corresponding number of options. For example, if the GUI 16 
were programmed to have 11 different states, then each node of 
the decision tree would have 11 options instead of the three 
options at each node shown in the decision tree of Figure 4. 

Referring to the decision tree of Figure 4, block 70 

20 of the decision tree corresponds to block 40 of the flow 

diagram of Figure 3. At this point, the microprocessor 
presents the initial situation or scenario on the display 10. 
Then, in this example, the user can indicate one of three 
attitudes to the initial situation, namely negative, neutral 
or positive. If the user has a positive attitude to the 
initial situation and adjusts the GUI 16 to reflect his or her 
positive attitude, then the microprocessor will cause the 
situation or character 14 to react accordingly, as shown at 
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block 72. If, in response to the character's reaction, the 
user now has a negative attitude and adjusts the GUI 16 to 
reflect his or her negative attitude, then the microprocessor 
will cause the situation or character 14 to react accordingly, 
as shown at block 74. If, in response to the character's 
reaction, the user now has a neutral attitude and adjusts the 
GUI 16 to reflect his or her neutral attitude, then the 
microprocessor will cause the situation or character 14 to 
react accordingly, as shown at block 76 . 

10 It is possible that the microprocessor can be 

programmed to react uniquely for each different block depicted 
in the decision tree of Figure 4. In that case, at any 
particular step or block in the decision tree of Figure 4, if 
the user has a positive attitude (for example) instead of a 
negative attitude (for example) , then all the remaining 
situations in the program will be different (from what they 
would have been if the user had a negative attitude at that 
point in the program). On the other hand, the microprocessor 
can be programmed so that it is possible to arrive at the same 

20 situation from a different sequence of prior attitudes. For 

example, the microprocessor could be programmed so that the 
same situation results from blocks 76 and 78 even though they 
each represent a different sequence of attitudes to previous 
situations. 

A partial program listing is attached as an 
Appendix. The program listing shows one possible 
implementation of an embodiment of the present invention using 
the programming language "Lingo" . 
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Numerous modifications and variations of the present 
invention are possible in light of the above teachings. It is 
therefore to be understood that within the scope of the 
appended claims, the invention may be practised otherwise than 
as specifically described herein. For example, the 
microprocessor could be programmed to consider other emotions 
of the user alone or in addition to the user's attitude. In 
the embodiment described above, left and right movement of the 
cursor 20 over the avatar 17 reflects a positive and a 

10 negative change in the user's attitude, respectively. Other 

options could be programmed. For example, movement of the 
cursor 20 towards the top of the computer display 10 could 
reflect the user becoming happier while movement of the cursor 
2 0 towards the bottom of the computer display 10 could reflect 
the user becoming increasingly sad. 

Another possible modification to the embodiment 
described above is to allow the user to select one of a number 
of different avatars. For example, possible avatars could 
include an attractive woman, an old man or a muscular soldier. 

20 Another possible avatar may be scanned images of the user him 

or herself, which could be incorporated as part of the 
computer program from scanned photographs of different 
expressions of the user. The characters in the computer 
program may be programmed to behave differently in a given 
situation to a particular avatar. For example, at a given 
point in the computer program, a character may react 
differently to an attractive woman with a negative attitude 
than to an old man with a negative attitude. 
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As another possible modification to the embodiments 
described above, between blocks 62 and 64 of Figure 3, an 
additional element of the program could be incorporated 
involving verbal responses by the user. In this case, the 
program would incorporate voice response and recognition 
software programmed to recognize particular key words. For 
example, if a character said something causing the user to 
have a positive attitude, the user would adjust the GUI 16 
accordingly and the user might say "yeah, right" (meaning, 
perhaps "yes, I agree") . If the microprocessor were 
programmed to understand the key words "yeah, right" in this 
situation, the character would then react accordingly. This 
situation is to be contrasted to a situation where the user 
having a negative attitude says "yeah, right" (which, in the 
context of a negative attitude would likely be sarcastic and 
suggest "I don't believe you"). Given the user's attitude, 
the program would construe the meaning of the expression 
"yeah, right" differently, and the next situation would 
proceed accordingly. If the program is modified to include 
this speech recognition element, the decision tree in Figure 4 
would, of course, have to be modified accordingly to 
incorporate different additional outputs from each node of the 
tree depending upon the user's attitude to any particular 
situation in addition to the user's verbal reaction. 
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THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE 
PROPERTY OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS : 

1. A computer program for use in a computer apparatus, 
the computer apparatus having a visual display device, an 
audio device and an input means, wherein the computer program 
when implemented on the computer apparatus displays on the 
display device a graphical user interface having a changeable 
appearance and wherein, in response to information conveyed by 
the computer program on at least one of the display device and 
the audio device, a user, using the input means, sets the 
appearance of the graphical user interface to correspond with 
an emotion of the user and wherein in response to the emotion 
of the user as represented on the graphical user interface, 
the computer program conveys further information on at least 
one of the display device and the audio device . 

2 . The computer program of claim 1 wherein the 
graphical user interface comprises an avatar. 

3 . The computer program of claim 2 wherein the emotion 
of the user as represented on the avatar is the attitude of 
the user. 

4. The computer program of claim 3 wherein the avatar - 
is a face. 
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5 . The computer program of claim 4 wherein the input 
means comprises a pointing means. 

6. The computer program of claim 5 wherein the pointing 
means comprises a computer mouse and a cursor displayed on the 
display device . 

7. An interactive, electronic apparatus for a user to 
communicate the emotion of the user comprising 

a display screen; 
an audio device; 

a graphical user interface visible on the display 
screen and having a changeable appearance; an 
input device for the user to set the appearance 
of the graphical user interface to correspond 
with an emotion of the user; 

a computer program implemented on the interactive, 

electronic apparatus, for conveying information 
on at least one of the display screen and the 
audio device and wherein in response to the 
emotion of the user as represented in the 
appearance of the graphical user interface, the 
computer program conveys further information on 
at least one of the display screen and the 
audio device. 

8. The interactive, electronic apparatus of claim 7 
wherein the graphical user interface comprises an avatar. 
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9. The interactive, electronic apparatus of claim 8 
wherein the emotion of the user as represented in the 
appearance of the avatar is the attitude of the user. 

10. The interactive, electronic apparatus of claim 9 
wherein the avatar is a face. 

11. The interactive, electronic apparatus of claim 10 
wherein the input means comprises a pointing means. 

12. The interactive, electronic apparatus of claim 11 
wherein the pointing means comprises a computer mouse and a 
cursor displayed on the display screen. 

13 . A method for a user to interact with a computer 
program run on a computer wherein the computer comprises a 
display means, an audio device and an input means and the 
computer program displays a graphical user interface having a 
changeable appearance on the display means, wherein the method 
comprises the steps of 

the computer program conveying information on at 
least one of the display means and the audio 
device ; 

the user, using the input means, setting the 

appearance of the graphical user interface to 
correspond with an emotion of the user; and 

the computer program conveying further information 
on at least one of the display means and the 
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audio device in response to the emotion of the 
user as represented on the graphical user 
interface . 

14 . The method of claim 13 wherein the graphical user 
interface comprises an avatar. 

15. The method of claim 14 wherein the emotion of the 
user as represented on the avatar is the attitude of the user. 

16. The method of claim 15 wherein the avatar is a face. 

17. The method of claim 16 wherein the input means 
comprises a pointing means. 

18. The method of claim 17 wherein the pointing means 
comprises a computer mouse and a cursor displayed on the 
display means . 

SMART & BIGGAR 
OTTAWA , CANADA 

PATENT AGENTS 
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ABSTRACT 

A graphical user interface for a computer program 
includes the representation of a face (an avatar) on a 
computer screen which allows a user to communicate the user's 
attitude to a situation represented by the computer program. 
The expression of the face changes according to the user's 
movement of a cursor over the face. In response to a 
situation appearing on the computer screen, the user sets the 
expression on the face to correspond with the user's attitude. 
The situation on the computer screen changes accordingly. 



